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Foreword 



As STATE AFTER STATE, 1111(1 locality 
after locality, turns to the implemen- 
tation of a programtnmg-planning- 
budgeting (PPB) system, educators, 
both administrators and teat Iters, and 
tilt* lay public- -school board members, 
state legislators, and other interested 
citizens' — need to lie aware of these de- 
velopments and their implications for 
schools. 

The NEA Committee on Educa- 
tional Finance sponsored the prepara- 
tion of this document so that educa- 
tors and citizens would focus on a PPB 
system consistent with the PPB system 
developed for other state and local 
government services by the State-Local 
Finances Project of The George 
Washington University under the di- 
rection of Dr. Selma J. Mushkin. This 
document, drawing as it does on the 
earlier experience and research studies 
by members of the project staff, pro- 
vides a basis for a dialogue between 
educators and others implementing a 
PPB system. 

The Committee on Educational Fi- 
nance, furthermore, seeks to bring 



these developments in general gov- 
ernment and school administration to 
the attention of classroom teachers 
and curriculum specialists because il 
a PPB system is to ojmrate ellectively 
with attention to products, process, 
and resources, the expertise of class- 
room teachers and curriculum special- 
ists must be brought to bear. As this 
document points out, there is a need 
for discussion among educators to as- 
sure contributions of these members 
of the profession as well as of the 
planning and budgeting officials in 
the planning for educational develop- 
ment. 

This document hopes to provide the 
framework within which PPB systems 
can develop for schools through co- 
operation of all those interested in 
furthering an integrated planning- 
programming-budgeting system for 
education as well as for other govern- 
mental services. 

William D. Firman, Chairman 

NEA Committee on Educational 
Finance, 1967-68 
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Planning for Educational Development in a Planning, 
Programming, Budgeting System 

Selma Mushkin and James R. Clcaveland 

I. What Is a PPB System? 



THE PRESENT PAPER is prepared 
in response to the frequently posed 
question of legislators, school admin- 
istrators, and school boards: 

What is a PPB system and what does 
it mean for education ? 

Planning, programming, budgeting 
(PPB) as an integrated system is gain- 
ing widespread use because of the 
growth in public expenditures and 
the consequent urgency of asking: 

How better — more effectively and effi- 
ciently — can available resources be 
allocated among competitive uses? 

Tax resources and the resources of 
men and material needed in support 
of public services are clearly not un- 
limited, and the demands on those 
resources are many. Education, as a 
major component of public expendi- 
tures, now claims a principal share of 
tax resources. And in view of the role 
assigned to education in meeting cur- 
rent economic and social problems, 
further expansion in educational 

Prepared by the State-Local Finances Project, 
The George Washington University, for the 
SEA Committee on Educational Finance. 

Dr. Mushkin is Project Director, and Mr. 
Cleaveland is Research Scientist. 



spending is projected . 1 In addition 
to economy and efficiency in program 
operations, current national problems 
point also to concern for greater 
equity in the use of tax resources for 
the attack on poverty and the educa- 
tion of the disadvantaged. 

Public officials, school boards, school 
administrators, teachers, and parents 
all contribute to school policy. Each 
of these groups is concerned with the 
public product produced — its quan- 
tity and quality — by the school and 
its progress toward satisfying the pur- 
poses of educational spending. 

Implicit in a PPB system is a con- 
tinuing process of review and analyst 
of all programs and activities of a 
school system. On-going programs as 
well as new programs are considered. 
This review of the total operation in 
line with the current objectives of the 
schools is perhaps the best defense 
against obsolescence. A PPB system 
can help to facilitate better public de- 
cisions on use of resources by provid- 
ing policy makers, whether at the 

1 Mushkin, Selma J. ( and McLoone, Eu- 
gene P. Local School Expenditures: 1970 
Projections. Chicago: Council of State Gov- 
ernments, 1965. 84 p. 



legislative or executive level, with 
the information required to sort out 
choices and to help define the choices. 
While PPB has been addressed largely 
to executive decision making, the 
processes of PPB can also be applied 
to facilitate the critical decisions of 
the legislators. 

PPB and the legislature — Imple- 
mentation of a planning, program- 
ming, budgeting system in the execu- 
tive branch of the government would 
provide legislators with more data 
about program proposals submitted 
to them and better documentation of 
by ’get ary requests. It thus would 
provide legislators with the informa- 
tional materials required for a greater 
understanding of the policy problems 
and the alternative approaches to 
them. The legislatures may have an 
independent analysis staff. Such a 
unit would aid in review and assess- 
ment of the informational materials 
provided to the legislature by the 
executive. It would permit careful 
evaluation of program trends and en- 
able the legislators to propose effec- 
tive alternatives to those submitted 
by the executive departments and 
agencies. 

Federal encouragement — The grow- 
ing need for program planning and 
evaluation as a management tool has 
encouraged states and localities, as 
well as educational authorities, to 
begin the process of implementing 
such a system. This encouragement 
has been augmented by the emphasis 
of several of the recent federal aid 
programs on planning and on pro- 
gram evaluation. The Elementary 
and Secondary Education Act, for 
example, now seeks to encourage in- 
tegrated planning of education, the 
development of objective measures of 
program consequences, and the test- 



ing of programs in terms of these ob- 
jective measures. 

An intergovernmental demonstra- 
tion— Five states, five cities, and five 
counties have undertaken to work to- 
gether through the initial stages of a 
PPB system and to reassess intergov- 
ernmental program problems within 
this framework. This 5-5-5 Project, 
financed by a grant to The George 
Washington University fron the Ford 
Foundation, as a demonstration, is 
providing information on the proc- 
esses, the potentials, and the issues 
involved in instituting an integrated 
system for planning public services 
and the facilities to provide them. 
The states participating in this dem- 
onstration are California, Michigan, 
New York, Vermont, and Wisconsin; 
the cities an,; Dayton, Denver, De- 
troit, New Haven, and San Diego; the 
counties are Dade (Fla.), Davidson 
(Tenn.), Los Angeles (Calif.), Nassau 
(N.Y.), and Wayne (Mich.). 

A PPB System Defined 

A PPB system is an integrated sys- 
tem to provide executive and legis- 
lative officials with better and more 
information for planning programs 
and for making choices among the 
optional ways funds can be devoted 
to achieve governmental objectives. It 
aids the decision-making processes in 
helping find new ways, through analy- 
sis and evaluation of public programs, 
of doing the public business faster, 
better, and less expensively. 

Analysis and program evaluations 
that are central to the implementation 
of a planning, programming, budget- 
ing system require identification of 
the public products that society de- 
sires. Such identification of public 
goods and services marks a shift in 
emphasis. The question of whether 
there should be 30 or 20 pupils per 
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teacher is replaced, for example, by 
now much of what type of learning 
do we want in our schools. Analysis 
reqi ires that activities be considered 
as they relate to each other, some com- 
plementary, some competitive. The 
search for alternative ways of meeting 
defined objectives makes PPB a so- 
cially innovative system. Also con- 
sidered are the optional ways of com- 
bining, for example, teaching staffs, 
teacher aids, facilities, and teaching 
materials, to effect the product de- 
sired. The important question rou- 
tinely asked in the course of imple- 
mentation of a PPB system is: 

How much additionally would he 
gained or lost by way of achiet ng the 
defined objective through spending 
more or less for the purpose? 

Better staff analysis and program 
evaluation will reduce some of the 
pressures on public officials that origi- 
nate in failure to consider program 
consequences ahead of time. 

An integration of familiar tech- 
niques— Within a PPB system the fa- 
miliar processes of program develop- 
ment and of budgeting are explicitly 
combined. It is a system in the sense 
of centering on program analysis in 
the light of explicitly defined objec- 
tives. It calls for program plans that 
can carry out these purposes and for 
budgetary requests that can help im- 
plement the planned programs. It is 
a system, too, in another sense, in that 
it calls for the identification of all 
activities that relate to the achieve- 
ment of a defined objective. For ex- 
ample, it calls for the identification 
of the range of programs that are a 
part of achieving learning in the 
amount and of the type desired, even 
though not all of these programs are 
carried out by the traditional “educa- 
tional” agencies. 



The newness of PPB arises not from 
the individual techniques that have 
been developed but from their inte- 
gration into a system and their pro- 
cedural application to government 
decision making. These techniques 
have zeroed in on the preparation of 
a series of documents that are major 
tools of a PPB system. They are enu- 
merated here: 

1. The program structure and state- 
ment of objectives 

2. Program analyses (cost-effective- 
ness analyses) and memoranda 

3. The multi-year program and fi- 
nancial plan. 

In the preparation of these docu- 
ments the PPB system requires: 

1. Clarifying and specifying the ul- 
timate goals or objectives of each 
activity for which a government bud- 
gets money 

2. Gathering contributing activities 
into comprehensive categories or pro- 
grams to achieve the specified objec- 
tives 

3. Examining as a continuous proc- 
ess how well each aciivity or program 
has done — its effectiveness — as a first 
step toward improving or even elimi- 
nating them 

4. Analyzing proposed improve- 
ments or new program proposals to 
see how effective they may be in 
achieving program goals 

5. Projecting the entire costs of 
each proposal not only for the first 
year but for several subsequent years 

6. Formulating a plan, based in 
part on the analysis of program cost 
and effectiveness, that leads to imple- 
mentation through the budget. 

Statement of Objectives and 
Program Structure 

The program structure reflects de- 
cisions on the goals und objectives 
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that are being sought through the 
government. Essentially it represents 
a classification of activities ancl pro- 
grams in accord with the goods ancl 
services produced to achieve the de- 
fined objectives. Formulation of basic 
objectives, goals, missions, or purposes 
thus becomes an initial phase of im- 
plementation of a PPB system. Such 
goals or objectives are not value free. 
On the contrary, they reflect the value 
judgments of those who represent the 
public. There is, accordingly, no sin- 
gle perfect model of a program struc- 
ture that encompasses all functions. 
Indeed, there are many ways of classi- 
fying programs even for a single set of 
defined objectives. 

The program structure is intended 
to highlight the basic objectives ancl 
to display the competing ana comple- 
mentary activities involved in carry- 
ing out such objectives. It provides 
the framework for analyzing programs 
and for the preparation of a multi- 
year program and financial plan. 

Essentially the defining of objec- 
tives of a governmental jurisdiction 
provides answers to a set of questions 
such as the following: 

What needs doing and forxvhom? 

Why is each activity currently per- 
formed being performed? 

The program structure itself dis- 
plays a hierarchy of program activi- 
ties. At the topmost levels are the 
broad categories that reflect the pro- 
grams designed to achieve the funda- 
mental objectives of the government. 
The second and lower levels display 
progressively narrower groupings serv- 
ing more limited objectives. As the 
categories become narrower in scope 
at the lower levels, the structure sets 
forth the complementary an a /or sub- 
stitute approaches to the fundamental 
objectives at the highest levels in the 



structure. The low’est levels of any 
structure would be composed of activi- 
ties and programs that are intended as 
specific means for moving toward the 
larger objectives. 

The display of programs in relation 
to objectives provides information on 
programs different in some ways from 
that contained in budget documents 
now submitted to legislatures. In 
many cases, budget documents are 
“line” item budgets rather than pro- 
gram-oriented budgets; fidl program 
costs do not always appear in the same 
budget category. 

Implementation of a PPB system 
does not require that the budget for- 
mat be altered, but it does require 
that for programming purposes ex- 
penditures be grouped in terms of 
program objectives rather than in 
terms of the items bought. In some 
jurisdictions budget formats have 
been altered. 

A line item budget may show the 
following items with little regard to 
the purposes being served: 

Salary ancl wages 

Contractual service, supplies, ancl 
materials 

Equipment purchases by type 

Employees’ retirement 

Workmen’s compensation. 

Such a listing conveys little infor- 
mation about why employees are on 
the public payroll, why supplies are 
purchased, or what groups in the 
population benefit from the outlays. 
If in addition to, or in place of, such 
line item information, a display shows 
the amounts expended in relation to 
objectives, those officials concerned 
with program policies W'ould have be- 
fore them information on expendi- 
tures according to purposes, for ex- 
ample: 
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Job training and placement 
— Formal vocational training 
— On-the-job training 
— Occupational counseling 
— Employment exchange informa- 
tion 

If in addition to such figures on 
expenditures, data are presented on 
program outputs that serve to display 
the volume or quality of the services 
provided, an additional range of sup- 
portive materials is given the decision 
maker. He would have set forth be- 
fore him the number of persons be- 
ing trained for jobs, receiving formal, 
school training, participating in on- 
the-job training or in joint school- 
industry training, assisted in selecting 
training, receiving employment infor- 
mation, and those who on completion 
of their training are employed at 
specified earnings levels. 

Program Analysis — Costs and 
Benefits in Alternatives 

A PPB system is a unifying and 
comparing process. On the one hand, 
consequences are assessed in terms of 
costs, both those that are immediate 
and those that are implicit for subse- 
quent periods as a result of immediate 
action. On the other hand, they are 
assessed in terms of benefits or pro- 
gram effectiveness. The arraying of 
cost and effectiveness for various pro- 
gram alternatives provides informa- 
tion required for decision. 

Analysis essentially involves a re- 
duction of complex problems into 
their component segments so that 
each segment can be studied. Ques- 
tions of fact can be subjected through 
this process to the test of observed 
experience. Those aspects of the prob- 
lem that involve value judgment can 
be separately identified and the basis 
of the judgment made explicit. 



On the one hand, a cost-effective- 
ness analysis may use, if applicable, 
many of the techniques of mathe- 
matics, operations research, econom- 
ics, etc. On the other, cost-effective- 
ness analysis may require no more 
technical sophistication than the pull- 
ing together of already existing data 
in a meaningful and informative way. 
Analyses may also draw upon various 
technical and nontechnical studies 
previously done which are relevant. 

Recommendations made on the ba- 
sis of analysis within the procedures 
of a PPB system are presented in 
policy papers termed “program memo- 
randa:’ The “program memo” is a 
document covering one major pro- 
gram area or a major portion of a 
major program area. Its purpose is 
to present major program policy find- 
ings, specific recommendations, and 
the reasons for these recommenda- 
tions, including a summary of the 
analyses that have been made. It is 
submitted prior to detailed budget 
preparation. 

Linking Planning and Budget Deci- 
sions (The Multi-Year Program 
and Financial Plan) 

A PPB system makes long-range fis- 
cal planning a routine for govern- 
ment. The analysis of programs that 
provide the information for decision 
includes an examination of effective- 
ness over a period ahead and of cost 
implications for that future period. 
Programming is thus given a time per- 
spective that adds to the informational 
base available to the decision maker. 
A summary of the decisions taken on 
public outputs and costs, at first ten- 
tatively and later for legislative bud- 
get action, is included in the Multi- 
Year Program and Financial Plan, one 
of the principal documents of a PPB 
system. It consists of two parts: 
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1. A multi-year financial plan that displays 
the estimated funding required (including 
both operating and capital requirements) for 
each program category for each of a number 
of years (usually about five) into the future 

2. A multi-year program plan that is es- 
sentially a descriptive table displaying se- 
lected characteristics of each program cate- 
gory in order to indicate the size and scope 
of the program and, wherever possible, its 
expected accomplishments. 

The multi-year program and finan- 
cial plan is of particular importance 
at two major points in the program 
planning cycle. 

1. The first major function of the 
plan is in the program analysis phase 
of program planning. A tentative 
summarization of all initial individual 
program decisions provides the basis 
for an examination of the fiscal feasi- 
bilitv of the over-all decisions. For 

4 

example, it is likely that the tentative 
set of individual program decisions, 
when summarized in a multi-year fi- 
nancial plan format, will indicate a 
requirement for funds that is out of 
reach either in the current budget or 
in subsequent years. Revisions of the 
program decisions that make up the 
first tentative summary would then be 
needed. 

Such a tentative summarizing does 
not necessarily imply that programs 
should be planned only to the extent 
that their revenue sources are cur- 
rently authorized. The governmental 
jurisdiction may choose to plan for a 
level of expenditure exceeding that 
which can be raised under current 
revenue provisions. But the question 
of revision of the revenue structure 
will be clearly posed. 

2. The plan plays its second major 
function after the budget has been ap- 
proved by the legislative body and a 
revised multi-year plan is prepared to 
reflect the latest decisions. The plan 
then becomes the “base,” or “baseline” 



displaying the jurisdiction’s current 
multi-year plan. As such, it would 
provide “planners” at all levels with 
a perspective as to what the organiza- 
tion is doing and where it is headed. 
It also becomes the composite pro- 
gram “plan” to which future program 
proposals should be compared. 

The Centralization or Decentraliza- 
tion of the Procedures 

In some instances state governments 
have placed initial responsibility for 
developing objectives and the output- 
oriented categorizations of activities 
on departments and agencies. In 
others the initial dialogues with policy 
officials to define these objectives have 
been conducted by central staff. In 
almost all local communities the ini- 
tial identification of goals and objec- 
tives ha*- been the responsibility of a 
central stall to the mavor or countv 

4 • 

manager. An integrated system, if it 
is to [>erform as a system, points to 
central staff participation in the form- 
ulation of public service objectives. 
Classification of activities by purpose 
within the basic objectives similarly 
requires identification and grouping 
of all activities that serve a common 
objective even though the activities 
are administered by different agencies. 

Qualifications of PPB as a tool — 
PPB cannot provide the answer to 
every question which an official may 
want answered before making a deci- 
sion. Responsible decision making 
will continue to rest on political judg- 
ment. And it cannot deal with all 
government’s problems. It does not 
address such problems as assessment 
of work efficiency, manpower selec- 
tion, or administrative procedures. 

PPB and analyses of costs and effec- 
tiveness of public programs can help 
by facilitating better decisions. The 
thrust of the system is to encourage a 
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conscious process of choice, sup]>orted 
by an adequate factual base. The pro- 
ductiveness of systematic analysis de- 
jxmuIs u|K)n the policy maker who uses 
it. whether at the legislative or at the 
exec utive level. 



In short, a developed PPB system 
will make timer information avail- 
able so that government may liecome 
not only more efficient but also more 
resjxmsive to the iw.hK and desires of 
the people. 



II. “Learning” as a Product of Government 



WE ASK IN THIS SECTION: 

How may educational activities be 
grouped within a government-wide 
program structure in an integrated 
planning, programming, budgeting 
system / What are the key objectives 
of education i What are the purposes 
of specific programs carried out by 
school systems? 

There is no "right" way of defining 
the "objectives" either of education or 
of school systems, but we have as- 
sumed "learning" as the central pur- 
pose of education and the schools. 

Public education claims the largest 
single part of the civilian tax dollar. 
This priority is no accident; it reflects 
the importance society places on the 
unique learning capacity of people 
and on transmitting its values and 
stock of knowledge. 

In what follows we show how the 
objective learning might fit into a pro- 
gram structure of government and 
then show within this objective a pos- 
sible classification of programs which 
promote the many purposes of learn- 
ing. In this way we display a frame- 
work within which to make more in- 
formed policy choices for education. 

Products or services produced by 
government may be classified in ac- 
cord with many differing themes. A 



jurisdiction may, for example, take 
as its major theme the development 
of man’s jx>tential as underscored in 
a statement by Professor Kenneth 
Uoulding of University of Colorado: 

It must lU'M i Ik* forgotten that the ultimate 
thing which am society .s producing is 
jK*ople. All other things are intermediate 
gcKKls. and all oigani/ations are intermediate 
organizations. No matter how rich we arc 
or how jKiwerful we are. if we do not pro- 
duce people who can at least lK*gin to expand 
into tin* enonnous potential of man. the 
MKietv must he adjudged a failure. 

Development of Man’s Potential 

The formulation of objectives and 
grouping of public programs in ac- 
cord with a selected theme, such as 
"Development of Man’s Potential," 
may be defined further by structuring 
programs in terms of the age groups 
in the population to which they are 
directed as shown in the following 
example: 

Theme: Development of Man’s 
Potential 

I. Developing Children m Their 
Early Years 

II. Developing Children and Youth 
A. Developing intellectual ca- 
pacity 
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B. Maintaining and improving 
physical and emotional well* 
l>eing 

C. Maintaining and improving 
family environment 

1) Maintaining and improving 
neighljorliood environment 
E. Correcting malbehavior and 
protecting the public 

III. Realizing the Potential of Work- 
ing-Age Groups 

IV. Realizing the Potential of the 
Aged 

v'. General Support and Adminis- 
tration 

We have listed a few illustrative 
classes of program objectives within 
the category, “Developing Children 
and Youth," the primary age group 
for whom formal education is carried 
out; for other age groups in the popu- 
lation, similar subcategories ait. called 
for in elaborating the range of pro- 
grams of governments. 

Public Investment for 
Economic Growth 

Another formulation of a central 
theme is that of public investment for 
economic growth in which the range 
of public investment is set forth in 
accord with the various factors con- 
tributing to such growth, including 
human resources, physical capital re- 
sources, and natural resources. All gov- 
ernment activities might be grouped 
as follows: 

Theme: Public Investment for 
Economic Growth 

I. Investment in Human Resources 

A. Health 

B. Learning 

II. Economic Growth — Industrial 
Supports 

III. National Resource Conservation 
and Development 



IV. Promotion of Community Devel- 

opment 

V. Environmental Improvements 

and Safeguards 

VI. General Administration and Suj>- 

j)ort 

In this grouping, learning is one of 
two major products that contribute 
to human resource investment as a 
source of economic growth; good 
health is the other. Because of the 
iinpjrtante of economic growth to the 
over-all resources available for public 
and private use and enjoyment, em- 
phasis is frequently placed on invest- 
ment, both in people and in things. 
Education’s major role as a source of 
growth is now widely accepted. Muc h 
weight is given to this investment role 
in designing educational programs. 

Consumer Services 

Still another illustrative theme 
views governmental services as a se- 
ries of consumer goods. In fact, the 
application of the processes of a plan- 
ning, programming, budgeting system 
to government is intended to provide 
analyses of goods that are comparable 
to the consumer’s evaluation of his 
options for spending his funds. The 
context of application of the processes 
of the system to alternative use of 
resources tends to emphasize this con- 
sumer services theme. 

Learning in this context is basically 
a consumer good which society pur- 
chases collectively with its taxes, 
rather than purchasing it on an indi- 
vidual basis in the market place. 

Theme: Fundamental Public 
Consumer Services 

I. Ptrsonal Safety 

II. Health 

III. Learning 

IV. Satisfactory Home and Com- 

munitv Environment 

/ 
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V. Economic Satisfaction and Em- 
ployment Opportunities 

VI. Satisfactory Leisure-Time Op- 
jxirtunities 

VII. Transportation-Communica* 
tion-Location 

VIII. C idler al Administration and 
Support 

We have not shown subcategories 
for the learning objective in this array 
of public goods. The subcategories 
are displayed, however, in a subse- 
quent section of this paper. 

Products of the School System 

Tims far the illustrations have been 
confined to showing the purpose, 
learning, in accord with various 
themes for grouping governmental 
programs by basic objectives. We 
turn now to the second question, 
namely, the products of the school 
system. 

Learning is surely the most impor- 
tant but not the sole product of the 
school system. How about the other 
services which a school provides? How 
might school health, lunch, commu- 
nity programs, and traffic controls be 
shown in such a structure? In short, 
how do we show the non-learning out- 
puts of the school system in a pro- 
gram structure for the whole of gov- 
ernment^ 

The illustration below shows how 
one might array these non-learning 
functions of schools by the final ob- 
jective which each promotes. We have 
taken as a starting point of the pro- 
gram structure the first seven cate- 
gories of the Public Consumer Ser- 
vices theme displayed above. The 
school may contribute to personal 
safety through provision of civil de- 
fense facilities, school safety patrols, 
and assistance to law enforcement 
agencies; the school may contribute 



to health through control of infec- 
tious diseases, school nursing, health 
screening, and school treatment pro* 
grams; etc. Under Item III would be 
found the chief outputs of schools 
which for reasons of simplification we 
delay discussing to the following pages 
which develop the activities and out- 
puts related to learning. 

The Program Outputs of Schools 

I. Personal Safety (Protection from 
Personal Harm and Property 

I ,oss) 

Community civil defense fa- 
cilities 

Safety patrols 

Assisf.'P'O to law' enforcement 
agencies 

II. Health (Physical and Mental 
Well-Being) 

Control of infectious and con- 
tagious disease 
School nursing 
Health screening programs 
School treatment programs 

III. Learning 

(Basic mission) 

IV. Satisfactory Home and Commu- 
nity Environment 

Schools as a community facil- 
ity . . 

Intra-community relations 

V. Economic Satisfaction and Em- 
ployment Opportunities 

School welfare programs 
(clothing distribution, etc.) 
School meals programs 
Career opportunities 

(included essentially in III 
above) 

School-work programs 

(included essentially in III 
above) 

VI. Satisfactory Leisure-Time Op- 
portunities 

Recreational services 
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VII. Transportation- Communica- 
tion-Location 
Traffic controls 
School transportation facili- 
ties 

Classifying Learning Outputs 

For convenience of discussion two 
routes to learning may h distin- 
guished: experience and education. 
Education in turn is carried out in- 
formally through public communica- 
tion and family and other personal re- 
lationships and formally through the 
school program. Since we are con- 
cerned here primarily with outputs 
of a public service, namely, educa- 
tional services, we shall concentrate 
on the latter. However, we must be 
mindful that throughout life, experi- 
ence will complement and often sub- 
stitute for the learning processes of 
formal education. V*c have tried to 
illustrate this relationship on Chart 1 
where the word experience is placed 
in a box with dotted sides. We have 
also shown informal education proc- 
esses in the same way. 

The chart shows how learning as 
one of the basic outputs of govern- 
ment fits into the consumer services 
theme above. It distinguishes the 
learning services provided to different 
age groups. 

In Chart 2, “Learning Through 
Formal Education” is subclassified 
further to group all the activities un- 
dertaken to achieve a learning prod- 
uct, according to three major types of 
learning that are being sought: (a) 

the development of basic skills, (b) 
the development of moral and social 
skills, and (c) the development of indi- 
vidual fulfillment skills. Educational 
activities directed to school-age chil- 
dren and youth can be divided into 
these three categories, each descriptive 
of fundamental objectives. 



The basic skills on the left are those 
skills necessary to the development of 
children and youth into independent 
and self-sufficient persons. In addition 
to skills involved in caring for one’s 
physical self there are the language 
arts, computational, and reasoning 
skills basic to economic independence 
capped off by specific employment 
skills. In the center box are “Moral 
and Social Skills” through the acqui- 
sition of which the individual is pre- 
pared for living with others, in the 
family, in small groups, and in society. 
Finally on the right are “Individual 
Fulfillment Skills” through which the 
individual gains the intellectual ex- 
perience and training that will per- 
mit him to identifv and realize his 

4 

own potential. Included within this 
range of skills are those services that 
permit an individual to receive advice 
and counseling that will buttress 
through information his ability to 
choose, and to acquire the necessary 
learning. 

Further Classification 

The special services required to pro- 
mote learning of (a) basic skills, (b) 
moral and social skills, and (c) skills 
leading to individual fulfillment 
among groups of children would be 
identified within a program structure. 
Such identification helps to flag the 
multiple layers of objectives that are 
being pursued, partly in the interest 
of efficiency and effectiveness, and 
partly out of concern for fairness or 
equity in distribution of public funds. 
The distinguishing of the categories 
presented in Chart 3 is not intended 
necessarily to suggest the desirability 
of separate programs for each of the 
groups but rather to list the target 
groups of children for whom services 
need to lie considered and evaluated. 
Further subcategories would be intro- 
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1. Program Categories Display 
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2. Learning Through Formal Education 




clucecl in practice in evaluating the 
services; for example, groups of men- 
tally retarded would be divided show- 
ing those that are educable; children 
from low-income families, rural or 
urban, would be classified to identify 
those from Spanish-speaking families, 
from American Indian families, and 
Negro and other ethnic groups. 

Learning in an Output Oriented 
Program Structure 

A composite of the illustrative cate- 
gories that have been discussed is 
shown in Chart 4. This chart com- 
bines in a single display the program 
categories and subcategories set forth 
earlier, and also indicates that inter- 
relationships between selected public 



services have to be considered because 
together they contribute to achieving 
a single public objective. 

We have carried through as an il- 
lustration the classification of a por- 
tion of a program structure — that part 
oriented to the product learning. In 
the preparation of such a structure 
each of the categories would need defi- 
nition so that expenditures made for 
each product could be identified and 
recorded. The format, moreover, pro- 
vides the basis for displaying pro- 
jected outputs and the costs that 
would be incurred in achieving them 
for a period of time ahead. 

There is no tidy and precise way to 
sort out where each specific activity 
that is undertaken in carrying out a 



18 



set of objectives belongs within a pro- 
gram structure. Activities and the ex- 
penditures made in carrying them out 
frequently may be grouped under 
more than one category. The need 
for definition is thus underscored. 

An Example of a Program 
Descriptive Statement 

To provide an example of an out- 
line of a descriptive statement on ob- 
jectives and definitions of educational 
services, we have followed through on 
the illustration given earlier. While 
the outline is not as comprehensive 
as would be needed for a fully opera- 
tive programming system, it does serve 
to show the kinds of information that 
would be displayed for the public 
officials. Chart 5 presents this ex- 
ample in a highly summarized form 
on pages 21-24. 



Summary 

We have tried in this section to pre- 
sent various ways in which the broader 
purposes of governments may be clas- 
sified in terms of their objectives, in- 
dicating first how learning, the objec- 
tive we have selected to emphasize, 
might fit into a program structure for 
government. We have also attempted 
to show by example how the activities 
of the school system would be grouped 
within a government-wide program 
structure. We nave elaborated fur- 
ther the groupings of various activities 
that contribute to learning, distin- 
guishing in general three types of 
learning products that are sought and 
the types of activities that contribute 
to each. 

The information that would be pre- 
sented in this type of display for pur- 



3. Target Groups identified 



Learning of 



Target groups, by 
learning problems 



Target groups, by 
socioeconomic status 



Basic skills 

Moral and social skills 

Individual fulfillment skills 



The “average” child 
The gifted child 

The physically handicapped child 
The emotionally disturbed child 
The mentally retarded child 



The child in low-income rural areas 
The child in inner city 
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4. Learning in an “Output” Oriented Program Structure 




* These groupings are not intended to represent separate activities but to flag groups that should be 
considered in evaluation of program options. 



20 



poses of illuminating the program de- 
cisions taken and funds requested is 
by itself a significant forward step. In 
making the selection to illustrate 
product orientation, we were not un- 
mindful that funds generally are not 
budgeted in terms of activities as we 
have outlined them, nor are we un- 
mindful of the fact that educational 



expenditures are not routinely ac- 
counted for in these ways. New esti- 
mates would be needed in order to 
quantify the resources devoted to each 
activity category shown. 

It needs to be emphasized in con- 
clusion, however, that the classifica- 
tion or grouping of educational activi- 
ties depends on objectives. 



5. Objectives and Definitions 
of Educational Services 

DEVELOPMENT OF BASIC SKILLS 

Objective: Sequential Development of independent and self- 
sufficient persons through educational levels 

Definition: Activities that contribute directly to developing ca- 
pacity needed for independent existence 

Skills in caring for one’s physical self 

Objective: To prepare the individual for self-sufficiency 

Definition: Activities that enhance the individual’s competence 
to take care of himself 
Physical fitness skills 
Physical education 
Health habit skills 

Health screening (hearing, mental retardation, sight) 
Referrals and case follow-ups 
Health education 

First aid and life-saving instruction 
Personal hygiene instruction 
Sex education 

Instruction in dangers of tobacco, alcohol, and narcotics 

Skills basic to independent living 

Objective: To develop the general skills that are pre-condi- 
tions for employment 

Definition: Activities that sequentially develop basic skills for 
independent living 
Language skills 

Reading instruction: 

English as a second language (when necessary) 

Tutorial and remedial reading (when necessary) 

Spelling 
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Elements of English courses 

Oral expression 

Grammar 

Composition 

Debate 

Computational skills 
Arithmetic operations 
Non-whole numbers — decimals, fractions 
Basic applied mathematics 
Reasoning skills 

Introductory natural sciences 
Basic behavioral sciences 

Introductory abstract reasoning — algebra, geometry 
Syllogism as dealt with in English courses 

Skills basic to specific employment 

Objective: To prepare the individual for specific employment 

Definition: Activities that develop special competencies for em- 
ployment 

Vocational skills development 
Vocational shops 
Business courses 
Technical courses 

Occupational training for groups with special health prob- 
lems (retarded, physically disabled, etc.) 

Work experience development 
Apprenticeships 
On-the-job training 
Neighborhood Youth Corps 
School-work release 

SOCIAL AND MORAL SKILLS 

Objective: To transmit society’s fundamental values in order to 
facilitate group living 

Definition: Activities that provide group living experience; that 
transmit knowledge of behavior required for inter-personal and 
inter-community adjustment; that give a sense of moral values to 
the individual; that check amoral and anti-social behavior 

Family living skills 

Objective: To transmit concepts of family living , 

Definition: Activities that reinforce the general pattern of 
home life and that provide a basis for acquiring group par- 
ticipation skills 
Homemaking 

Opportunities for one-to-one personal working and play rela- 
tionships 



Group participation skills 

Objective: To transmit moral precepts and concepts of group 
participation 

Definition: Activities that involve instruction and experience 
meant to facilitate group participation and individual func- 
tioning within groups 
Play and other group recreation 
School social activities 
Racial relations programs 
Clubs and other quasi-formal groups 

Societal participation skills 

Objective: To develop a knowledge and appreciation of the 
societal environment and how to participate in it 

Definition: Activities that are intended to expose the indi- 
vidual to his societal environment and to promote his partici- 
pation in it 
Civics 
Geography 

History — American and general 
Student government 
Trips to observe governmental activities 
Speakers representing government and community organiza- 
tions 

INDIVIDUAL FULFILLMENT SKILLS 

Objective: To achieve awareness of the range of human endeavor 
and gain full self-realization 

Definition: Activities designed to develop the individual’s under- 
standing of society’s achievements and appreciation of himself, 
his capacities and competencies, and to motivate him and °' ; ve 
him the opportunity for self-development 

Knowledge of society’s achievements 

Objective: To achieve an awareness of the range of human 
endeavor and its development 

Definition: Activities that transmit a knowledge and awareness 
of the achievements of mankind and the importance of thought 
and creativity 
General study of: 

History 

Social geography 

Arts and music appreciation 

Attendance at art and music events 



Intellectual expression and training 

Objective: To draw out and broaden individual capacity and 
talents 

Definition: Activities that (a) enrich the regular school pro- 
gram, (b) provide opportunities for creative expression and for 
elevating the dignity of the individual, and (c) develop leader- 
ship talents 

Arts and music (including performing arts) 

Dance — ballet, folk dance, etc. 

Drawing, etc. 

Creative writing 
Music performing 
Crafts and design 
Foreign languages 
Intermediate and advanced science 
Science and mathematics 
Science competitions 
Naturalist field trips 

Intermediate and advanced humanities and social sciences 
Advanced social science, sociology, economics 
Advanced humanities, advanced literature 
Competitive sports 
Interschool sports 

Intramural extracurricular sports (activity included in an- 
other category) 

Student government (activity included in another category) 
Formalized institutional structures to allow students to 
participate and lead in student self-government as con- 
trasted to social, recreational, and academic-interest club 
activities 

Supports for individual choice 

Objective: To provide information and aid to the individual 
and his parents to permit better choic of school program and 
of post-high-school opportunity that accords with capabilities 

Definition: Activities that provide counseling, testing, and 
guidance and that enhance and support the capacity of the 
individual to attain self-realization 

General pupil testing; testing of pupils with special prob- 
lems; testing of teaching methods 
Guidance for choice among courses, among colleges, and 
among vocations 

Guidance and counseling of the family 

Psychiatric or clinical psychological guidance and counseling 
“College discovery” 
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III. Cost-Effectiveness Analyses for Learning 



IN THE USE OF LIMITED tax 
funds, educators, legislators, and 
elected officials are faced with choos- 
ing among programs. Analysis pro- 
vides information comparing the cost 
and effectiveness of the several alterna- 
tive programs. The questions: Why? 
What? Where? How? For whom? .aid 
When? can be clarified and given 
specificity by careful analysis. 

Educational problems that could be 
illuminated run the gamut from de- 
tailed questions to the broad issues of 
compensatory public education for 
slum children and for other disad- 
vantaged groups. Analysis may shed 
light on the question: Where should 
we locate the new school facility? Or, 
How’ should an educational facility 
be designed to facilitate learning? It 
would yield more and better informa- 
tion to facilitate choices in scientific 
instruments or learning devices. At 
the other extreme, analysis may be 
directed to such broad policy issues 
as: What should comprise the school 
year? How should education be 
financed? 

While analysis of programs is cen- 
tral to the carrying out of a planning, 
programming, budgeting system, the 
evaluation and analysis of educational 
programs may be undertaken with ad- 
vantage whether or not such analysis 
is part of an integrated planning, pro- 
gramming, and budgeting system. 

The concepts of analysis and evalu- 
ation are hardly new to those con- 
cerned with education. It is somewhat 
of a departure, however, that in recent 
federal legislation the U.S. Congress 
has specified evaluation as a condi- 
tion of aid. Congress requires evalua- 
tion of educational programs of both 



the Office of Economic Opportunity 
and the Department of Health, Edu- 
cation, and Welfare “in order that 
cognizance may be taken of desirable 
changes which should be made in fu- 
ture perfecting legislation.” This is 
an active role in policy questioning 
for the nation’s number one legisla- 
tive body. It is a role that will prob- 
ably become increasingly important 
in all federal grant-in-aid legislation. 
For example, Title I of the Elemen- 
tary and .Secondary Education Act of 
1965 requires "provision for approved 
objective measurements of educa- 
tional achievement . . . for evaluat- 
ing at least annuallv the effectiveness 

o * m 

of the program in meeting the special 
educational needs of educationally de- 
prived children.” 

Impetus is thus given to legislative 
requests for evaluation of educational 
projects and their funding. Such eval- 
uations will yield a body of informa- 
tion of gre«* value for analysis of 
program options. 

What Is Analysis? 

Analysis, as defined more specif- 
ically in the paragraphs below, is a 
process of systematically asking rele- 
vant questions about full cost impli- 
cations and benefits of program al- 
ternatives to satisfy objectives and 
assembling information that bears on 
those questions. The questioning 
starts by defining program purposes 
or objectives and by asking what al- 
ternative courses may be followed in 
meeting those objectives. Analysis 
calls for estimating costs and gains 
in meeting those purposes by alterna- 
tive programs. It calls for inquiring 
about the uncertainty of those esti- 
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mates. It calls for documentation of 
the information that is brought to- 
gether to give greater precision to the 
“pros and cons” (gains and costs) of 
alternatives for meeting stated objec- 
tives by quantified description where 
|x>ssible and to qualitative statements 
where quantification is not possible. 

The question and objectives — Spe- 
cifically, analysis addresses itself first 
to: Are we asking the right question 
in terms of our objectives or purposes 
when we ask, for example, What ser- 
vices are available in a community 
for linotype training? Should we ask 
instead, Are young persons finding 
jobs in the fields for which they are 
trained? Or are we addressing the 
right question when we ask, How can 
we achieve a standard of 25 pupils 
per teacher? Instead in terms of our 
purposes we might ask, How’ best can 
we achieve the learning products we 
seek in our school system? 

By systematically attempting to 
identify the underlying purposes of 
education and the schools, a frame- 
work is set for the development and 
the documentation of relevant infor- 
mation that can aid in sharpening the 
issues for decision and in identifying 
the component considerations, includ- 
ing cost implications and likely conse- 
quences in satisfying the purposes 
sought. 

Alternatives — Analysis calls for a 
search for alternative ways of meeting 
the defined objectives, a questioning: 
What are the options? Wliat can be 
done in place of what we are now do- 
ing? A questioning process on options 
systematically carried out yields a 
range of possibilities for examination 
as to costs and gains. 

Through the search for alternatives 
there is developed a built-in frame- 
work for social invention. A frequent 
criticism of government is that it is 



slow to accept new ideas. The edu- 
cational community, among other 
groups carrying responsibility for 
public services, has frequently been 
charged with a lack of receptivity to 
new ideas. Adoption of analytical 
methods that routinely call for imag- 
inative creation of new programs and 
program options should help to over- 
come this lethargy. 

The product of analysis, with its 
emphasis on alternatives, would con- 
front policy officials w'ith a (Afferent 
kind of decision. In place of the fa- 
miliar Yes or No response to specific 
proposals, a range of choices would be 
provided, both in terms of activities 
and in ways of carrying them out to 
achieve defined objectives. Thus, 
analysis may mean a more difficult de- 
cision, since it does not present merely 
a Yes or No choice. 

Documentation — Analysis means 

documentation, that is, a written 
statement on what is used as data 
what is assumed, what logical se- 
quence is followed in the reasoning. 
The sources and meaning of statistics 
that are drawn upon would be set 
forth. Major assumptions that art; 
made to simplify the study would be 
set forth in detail. The logical se- 
quence that is followed in arriving at 
the formulation of the content of the 
study would be displayed. Limita- 
tions, uncertainties, and factors not 
considered would be clearly identified. 

Such documentation permits others 
to review and understand what has 
been done. For those public officials 
who are especially concerned with any 
policy question or public issue, the 
documents prepared provide a basis 
for careful examination of the de- 
tailed materials that underlie sum- 
mary statements of choices for action. 

Costs and gains — Analysis calls for 
two basic sets of information once ob- 



26 



i 

i 



jectives have been specified and the 
questions have been defined: (a) in- 
formation on costs of programs that 
represent alternative ways of meeting 
objectives, and (b) information on out- 
puts or effectiveness relative to the ob 
jectives of various program or activity 
options. 

The information that needs to be 
brought together on costs and effec- 
tiveness occurs on three levels: 

1. Cost and effectiveness in a given 
current period for each level of the 
program 

2. Future cost and effectiveness im- 
plications of present programs and 
alternatives for each level of the pro- 
gram 

3. Changes in cost and effectiveness 
that accompany changes in level of 
volume or quality of services pro- 
vided, both current and future peri- 
ods. 

Program Costs in Alternatives 

Costing of public services and pro- 
jecting costs for a future period are 
the more familiar components of the 
information that is brought together 
as part of an analysis. 

For costing purposes we need to 
know the total estimated cost of each 
program alternative that is being con- 
sidered, both initial cost and expendi- 
tures that are implicit for the future 
at the levels of services being ex- 
amined. For example, the current 
level of provision of kindergarten serv- 
ices would have to be priced out for 
the changing number of 5-year-olds 
and kindergarten services for a half- 
day session and a full-day session, both 
for the present number of 5-year-olds 
and the changing number expected. 

Total costs include expenditures 
for personnel, such as salaries and 
wages and fringe benefits. They in- 



clude directly identified costs associ- 
ated with an activity and that part of 
overhead that appropriately may be 
assigned to it. Building costs, equip- 
ment costs, and supplies that would 
be involved in the various program al- 
ternatives would be estimated whether 
borne by the schools or by some other 
agency. Costs borne by general reve- 
nues of the jurisdiction would be 
identified as well as costs borne by 
grants and by other taxing jurisdic- 
tions. 

We also need to know, however, 
which costs involved in the activity 
being considered are fixed and ac- 
cordingly would not change because 
of changes in level of activity and 
which costs are variable (i.e., costs 
that respond to alternations in quan- 
tity or quality of service). Such a sepa- 
ration provides the base for estimation 
of how costs would be modified, or 
what the marginal cost effect would 
be, by greater or lesser production of 
the services. 

The work that has been done over 
the years in developing the manual 
for educational accounts and the use 
of such accounts by the U.S. Office of 
Education for statistical reporting 
provide an important take-off point 
for most analyses. However, expendi- 
ture accounts specified in the manual 
do not necessarily yield the total cost 
data needed for examining in depth 
the cost of particular optional activi- 
ties. Nor is it necessary that account- 
ing records and definitions be revised. 
It would be sufficient to prorate ex- 
penditures as recorded for accounting 
purposes to derive cost estimates. 
These prorations can be made in ac- 
cord with reasonable allocation in- 
dexes designed to allocate expendi- 
tures among purposes. While this pro- 
cedure would lack the precision of 
detailed accounts, it would provide 
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adequate information for program 
analyses. Margins of error are toler- 
able. The major consideration is 
whether in arraying alternative meth- 
ods of getting a task done the figures 
are comparable from program option 
to program option. 

Measuring Effectiveness 

Measurement of gains, benefits, or 
effectiveness that would be achieved 
by each of the alternate ways of satis- 
fying the specified objective are far 
less familiar. Much of the discussion 
that follows is accordingly addressed 
to: How does one measure the effec- 
tiveness of an educational activity? 



In private industry, product output 
is a relatively simple thing to deter- 
mine. The output of a car manufac- 
turer consists of the number of motor 
vehicles of various types produced — 
trucks, buses, passenger automobiles. 
For public goods and services, includ- 
ing education, indicators of progress 
or units of output have not received 
much attention heretofore. Recent 
efforts at evaluation of public pro- 
grams and services give a new' signifi- 
cance to output measurement. 

Professor Jesse Burkhead, in his 
studies on educational expenditures, 
has illustrated in summary form the 
production process of education in a 



6. Input, Process, and Output Variables 
in the Educational Process 



INPUT VARIABLES 

(Land, labor, capital) 

Student time 
in the classroom 
?.t home 
extracurricular 

Personnel time 
administrative 
teaching 
clerical 
maintenance 
auxiliary 

Materials and supplies 

Buildings and equip- 
ment 



PROCESS VARIABLES 
(Current expenditure 
policies) 

Glass size 

Size of the school 

Teacher-pupil ratio 

Ratio of administrative 
and clerical personnel 
to students 

Use of personnel for 
guidance for remedial 
instruction 



OUTPUT VARIABLES 

(Benefits to the indi- 
vidual and society) 

Increased intellectual 
curiosity 

l 

Social adaptation 

Development of crea- 
tivity 

Increase in skills and 
earning ability 

Increased lifetime 
earnings 



Growth of informed 
electorate 

Increased national 
growth 

Source: Burkhead, Jesse. “A New Way To View the Educational Process.” Education in the 
States. Washington, D.C.: National Committee for Support of the Public Schools, 1966. p. 29. 
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manner somewhat analogous to that 
of industry. He shows the interplay 
of inputs and outputs as shown in 
Chart 6. This illustration, as Profes- 
sor Burkhead emphasizes, does not 
identify the whole range of possible 
variables in inputs, in process, and in 
outputs. 

The measures we seek are those that 
can quantify how well we are meeting 
specified objectives. This quantifica- 
tion, in turn, requires new concepts 
of measurement. Though the aims of 
education may be hard to quantify, 
so are other of our basic aims. In 
our democratic society we hold free- 
dom dear, yet measurements of free- 
dom have by and large not been de- 
veloped. We have established institu- 
tions, our court system, for example, 
to assure our basic democratic rights, 
but we do not have measures of how 
effectively the courts and the govern- 
mental system as a whole are protect- 
ing those rights. The illustration of 
freedom is cited to suggest that the 
problems of defining measures of ef- 
fectiveness for education, while diffi- 
cult, are not unique. 

What determines the yardstick that 
is used to assess whether satisfactory 
progress toward the achievement of 
objectives is being made? Several 
characteristics of such yardsticks need 
emphasis. The first of these clearly is 
the relevance of the measure to both 
the objective and the activity. The 
second is the completeness of the mea- 
surement, that is, the indicators of 
progress in combination should quan- 
tify all significant effects of an activity. 
A further characteristic is to promote 
simplicity as long as the indexes en- 
compass the major program accom- 
plishments. However, as we might 
expect, the availability of information 
and the potential for collecting new 
data will affect the indicators used. 



Objectives and criteria of evalua- 
tion — Clearly the measures selected 
are determined by the objective of the 
activity or program. Assume, for ex- 
ample, that as the sole but still vaguely 
defined objective of the school system, 
“equality of economic opportunity” 
had been selected. This vaguely stated 
objective may be given greater speci- 
ficity in a number of ways. For illus- 
trative purposes, again to indicate the 
relation between the selection of prod- 
uct measures and objective, we might 
reformulate the objective of equality 
of economic opportunity in opera- 
tional terms as achieving completion 
of a 12th grade level of education for 
all young persons. (At this level, the 
research of Professor Richard S. Eck- 
haus of MIT suggests individuals 
qualify for all but 10 percent of the 
jobs in the economy. 2 ) The most direct 
measure of output would then become 
the number and proportion of persons 
satisfactorily achieving the defined 
high-school program. The definition 
of the program would be set in terms 
of basic requirements for all but 10 
percent of job opportunities. An edu- 
cational program would be more effec- 
tive the closer it came to achieving 
high-school completion for 100 per- 
cent of each school generation. With 
this limited statement of the objec- 
tive, other educational activities, such 
as the objective of college preparatory 
work, would not be considered, nor 
would any other educational product 
that is not job related. 

Multiple purposes arid multiple 
measures — Education as suggested by 
the outputs listed in Chart 1 serves 
multiple objectives and thereby calls 
for multiple indicators of progress 

2 Eckhaus, Richard S. “Economic Criteria 
for Education and Training." Review of Eco- 
nomics and Statistics, May 1964. 
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toward meeting those objectives. The 
chart displays benefits to the indi- 
vidual and to society. It includes 
variables that measure current effects, 
and those that measure long-range 
consequences. The selection of vari- 
ables from among the multiple possi- 
bilities again depends upon the objec- 
tives that are given emphasis by the 
public officials in a community or 
state. 

Multiple indicators for single pur- 
poses — Even within a single defined 
objective, adequate measurement of 
effectiveness may require the develop- 
ment and use of multiple indicators. 

Elsewhere we identified for illus- 
trative purposes “the achievement of 
learning” as a central objective of 
education. This learning product, in 
turn, was classified into (a) learning 
of basic skills, (b) learning of moral 
and social skills, and (c) learning skills 
for individual fulfillment. For these 
subcategories and their component 
elements, a number of indicators of 
output are shown in Chart 7. The 
chart follows an illustrative program 
structure. The indicators shown also 
are illustrative, but generally feasible 
in the present state of the art of test- 
ing. While not all school districts or 
educational departments have readily 
at hand the information that is called 
for by those indicators, the data could 
be collected. 

The newness of undertaking to ana- 
lyze programs in terms of their prod- 
uct and effectiveness will necessarily 
require collection of new data. It will 
also require for some programs the 
development of new concepts for data 
collection. For example, the illustra- 
tive indicators shown include achieve- 
ment and attitudinal measures that 
are feasible with present testing in- 
struments, but are not used in many 
places. To improve the illustrative 



indicators new tests would have to be 
designed. 

In some cases more than one indi- 
cator is listed for a single activity in 
measuring the progress toward satis- 
fying the objective. In selecting indi- 
cators it is possible to use so many 
that the decision-making process will 
be overloaded. At the other extreme, 
important products (negative as well 
as beneficial) can be overlooked. 

Matching “effectiveness” and ac- 
tivity — Measures or indicators of effec- 
tiveness necessarily must be selected 
so that they are appropriate for the 
activity or program that is being ana- 
lyzed. At the same time, however, the 
measuring rods should provide in- 
formation on the relation of the ac- 
tivity to the broader purpose served, 
just as the categories of a program 
structure display elements and sub- 
grouping of programs that are prod- 
uct oriented, so the measurements of 
the output will be more or less de- 
tailed as indicated in Chart 7. 

We have grouped counseling ac- 
tivities, for example, on this chart as 
one of several types of counseling that 
have the purpose of giving support 
to the individual in choosing school 
programs and post-liigh-school oppor- 
tunities appropriate to developing his 
own capabilities fully. How may we 
measure the product of family coun- 
seling in facilitating the full develop- 
ment of an individual’s capability? 
The number of children and their 
parents given counseling services is 
one possible indicator of the volume 
of activity. Another might be the 
closely related measure of the num- 
ber of children referred to and “fol- 
lowed through” family counseling ser- 
vices. Clearly volume of services is an 
inferior indicator to such measures as 
school achievement, or work satisfac- 
tion and so forth. But counseling often 
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involves a long period before its re- 
sults can be assessed even when a di- 
rect relationship can be established 
between achievement and family 
counseling. To come closer to mea- 
suring the product, it may be possible 
to gather information on diagnosis 
and prognosis by time intervals of 
cases referred for family counseling 
services and to separate out and mea- 
sure differences in individual adjust- 
ments and use of capacity between 
“before” and “after” services, in cases 
that are amenable to substantial im- 
provement in a relatively short period. 
It may also be possible to conduct a 
controlled experiment to obtain in- 
formation on the differences between 
“services” and “no services” in a short 
period. On examination it may be 
found that a short-term indicator 
w’oidd tend to bias the evaluation for 
a number of reasons, both in terms of 
cost and in terms of effectiveness of 
program, so that other measures of 
effectiveness would have to be devel- 
oped. The purpose of this example 
is merely to illustrate the process of 
indicator selection. 

Effectiveness for target groups — 
Educational services, often provided 
under state constitutions that guaran- 
tee the right of all children to edu- 
cation, require a careful assessment 
both of “equality of opportunity” and 
of fulfilling individual promise for 
groups of children varying in such 
characteristics as age, sex, family in- 
come, location, physical condition, 
and intellectual ability. 

While objectives may be defined in 
the same way for all children, alterna- 
tive programs may have different ef- 
fects in yielding learning of (a) basic 
skills, (b) moral and social skills, and 
(c) self-fulfillment skills, depending 
upon the intellectual and physical 
condition of the child and his socio- 
economic group. 



At this time two groups of children 
of school age are of particular govern- 
mental concern: (a) children in core 
cities and in poor rural communities 
for whom compensatory educational 
services are needed in order to achieve 
the learning objectives sought and (b) 
gifted children in and out of core 
cities on whose full intellectual de- 
velopment depends the quality of the 
population needed to gain the scien- 
tific and technological advances that 
may compensate for our nation’s 
smaller number of people in the 
world political scene. Thus, the mea- 
sures used can be directed to assess 
effectiveness for target groups; for ex- 
ample, test achievement scores might 
be provided by sex, economic group, 
and so forth. 

Monetary measures of effectiveness 
— While the additional earnings that 
accrue to those who complete high 
school have become a familiar indi- 
cator of education’s product, our em- 
phasis is on non-monetary outputs. 
These non-monetary outputs are more 
in accord with the central objective 
“learning” used for illustrative pur- 
pose in this pamphlet. We decided to 
emphasize these non-monetary mea- 
sures for the following additional 
reasons: 

1. The translation of educational 
benefits into dollars has become too 
ready an approach to the effectiveness 
of education. 

2. Activities that make up the over- 
all educational program require more 
specific measures of effectiveness than 
are provided by the dollar additions 
to earnings. 

3. Education produces current 
“value” for children and their parents 
that is not measured by future incre- 
ments to earnings. (It must be borne 
in mind that more than one-fourth of 
a person’s life expectancy — now 66.8 

(Continued on page 38) 
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Number and percent of children 
meeting specified physical stan- 
dards 
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years for males at birth — is the period 
of childhood.) 

4. Population mobility frequently 
creates benefits or returns from a full 
period of preparatory education in 
places other than the communities in 
which educational services were re- 
ceived by an individual. 

5. Educational programs serve mul- 
tiple purposes, and these purposes, 
especially as they concern different 
target groups, would not be displayed 
fully if combined in a single dollar 
measure. 

In summary, the measure of added 
earnings does not sufficiently reflect 
the full benefits of learning. 

Almost all the non-monetary indi- 
cators could be translated in one way 
or another by assigning what is termed 
“shadow prices” or some monetary 
value. For example, an estimate might 
be developed of what parents would 
be willing to spend to have Johnny 
move from grade 4 to grade 5. 

There are, however, types of mea- 
sures of effectiveness appropriate to 
the objective selected that can readily 
and appropriately be put forth in dol- 
lar terms. We have already indicated 
the increments to earnings criterion 
which is applied frequently. There 
are other monetary measures such as 
the reduction in cost of noneduca- 
tional public services as a consequence 
of “learning.” These may include sav- 
ings in expenditures for welfare, ju- 
venile delinquency, health, and so 
forth. 

Relating Costs and Effectiveness 

Analysis of programs concerns itself 
with relating the cost of various op- 
tional means of achieving an objective 
to the gains or output which are in- 
dicated by the values estimated for 
the measure of effectiveness. Dollar 



changes in cost for each alternative 
program mix and level, when arrayed 
against the changes in effectiveness, 
provide the skeletal format for ana- 
lyzing programs or activities. In an 
appended statement we present a 
hypothetical example of an analysis 
to show, at least in a partial way, the 
process and the kinds of information 
that would become available. 

Qualification and Summary 

Analysis of educational programs is 
clearly difficult because of the special 
characteristics of the services pro- 
duced. Among these characteristics 
which make analysis difficult are: 

1. The long gestation period of 
education outputs and the length of 
the necessarily sequential learning 
processes. 

2. Our limited knowledge of the 
learning process which might hamper 
attempts to attribute a particular re- 
sult to the actual activity which pro- 
duced it. 

3. The multiplicity of objectives in 
education which complicates the task 
of assigning a particular activity to 
the final educational purpose which 
it serves. 

4. The difficulty of factoring out 
the effects of nonschool experiences on 
the process and product of learning. 

Allocation of tax funds for educa- 
tion requires information on the rela- 
tive effectiveness of alternative types 
of programs. They require costing of 
the various options. While frequently 
costs may be specified within a reason- 
able margin, prediction of effective- 
ness is often much more complex. 
Part of the difference lies in the fa- 
miliarity of cost estimation, and the 
unfamiliarity of effectiveness quanti- 
fication. As more work is done on 
program analysis and additional re- 
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sourc es are devoted to program evalu- 
ation and effectiveness measurement, 
the tasks of program analysis will 
become easier. 

Even now efforts are being made to 
assemble the findings of earlier ex- 
jHjrimental studies that could provide 
a “data base" for prediction of the 
effectiveness of various educational 
programs and activities. New experi- 



mental studies with adequate controls 
are being undertaken. A substantial 
enlargement of the research and pro- 
gram evaluation effort is indicated. 
Program analysis, however, and gen- 
eration of alternatives will likely re- 
main a creative art rather than a 
science — a creative formulation of the 
“problem 5 " and a creative develop- 
ment of means of attacking it. 



Appendix: An Illustrative Example of the Process 
of Cost-Effectiveness Analysis 



A BRIEF EXAMPLE is presented 
here to help clarify, at least in a 
summary way, the difference analysis 
would make in informational mate- 
rials presented to those faced with de- 
cisions on public questions.* 

For this example we selected child 
care (or supervisory) activities now 
peripheral to school operation in 
most communities. For simplicity’s 
sake only, we do not consider any 
learning effects of these child care 

* The assistance of Brian Herman, Re- 
search Scientist, in developing this example 
is gratefully acknowledged. 



services. Our defined objectives of 
child care as an illustration, are (a) 
to give mothers more choice in satis- 
factory employment, and (b) to en- 
hance family income. 

In outlining the processes we (a) 
generate a series of optional programs, 
(b) formulate measures of progress 
toward satisfying the defined objec- 
tive, (c) quantify the progress in terms 
of the measures of effectiveness, (d) 
assess the additional costs implied by 
the optional programs. 

Program options— We. consider sev- 
eral changes in school operations that 



TABLE 1 . — CHARACTERISTICS OF CHILD-CARE PROGRAM OPTIONS 



Program options 



Program characteristics 


I 


II 


III 


IV 


V 


1. Hours of service 

2. Weeks of service 

3. Home health services 


9 A.M.- 
3 P.M. 
40 

None 


8 A.M.- 
6 P.M. 
40 

None 


8 A.M.- 
6 P.M. 
52 

None 


8 A.M.- 
6 P.M. 
40 

Range 


8 A.M.- 
6 P.M. 
52 

Range 
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would broaden the choice of employ- 
ment outside the home with its added 
money income for mothers. These 
include: 

1. Lengthening the hours of school 
activities for children five years of age 
and over on an elective basis 

2. In addition to lengthening the 
hours per day, extending the number 
of weeks of school activities from an 
assumed 40 weeks to 52 on an elective 
basis 

3. During an extended day pro- 
viding a range of home health serv- 
ices (practical nurse, home making, 
etc.) to care for the child who is tem- 
porarily ill, either for 40 weeks or 
52 weeks. 

Again for ease of presentation we 
do not consider programs other than 
child care that would facilitate a 
broadened choice of satisfactory em- 
ployment for mothers,- nor do we con- 
sider reducing below five years the 



age of children for whom child care 
activities might be provided. 

We display in Table 1 the hypo- 
thetical characteristics of an assumed 
on-going program, and then vary the 
hours, weeks, and provision of home 
health services in several combinations 
to provide more supervised activities 
for the child of school age. 

Program I is the assumed on-going 
program. 

Program II is lengthening of the 
hours of school activities from 9 a.m.- 
3 p.m. to 8 a.m.-C) p.m. on an elec- 
tive basis. 

Program III is lengthening hours 
from 9 a.m.-3 p.m. to 8 a.m.-6 p.m. and 
in addition extending the weeks of 
school activities to 52 weeks, on an 
elective basis. 

Program IV is a provision of school 
activities from 8 a.m.-6 p.m. and also 
health services, for the 40 weeks of the 
on-going school year. 



TABLE 2 .— ILLUSTRATIVE EMPLOYMENT EFFECT OF CHILD-CARE PROGRAM 

OPTIONS (NUMBER OF MOTHERS) 









Program options 




Mothers’ employment status 


I 

On-going 

program 


II 

Added 

hours 


III 

Added 

hours 

and 

weeks 


IV 

Home health 
and added 
hours 


V 

Home health 
and added 
hours and 
weeks 


Remain at home 


600 


500 


450 


450 


410 


Are employed within hours 


9 A.M.-6 P.M. only 


100 


50 


40 


70 


60 


Are employed within hours 


8 A.M.-6 P.M. but not exclu- 
sively during 9 A.M.-3 P.M.. . 


240 


400 


460 


430 


480 


Are employed hours other than 


8 A.M.-6 P.M 


60 


50 


50 


50 


50 


Total 


1,000 


1,000 


1,000 


1,000 


1,000 


With improved choice 

Without improved choice * . . 


800 


800 


800 


800 


800 


200 


200 


200 


200 


200 



* Mothers with children below existing school age, or those not able to work outside the 
home. 
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TABLE 4.— ILLUSTRATIVE 


NUMBER 


OF PUPILS BELONGING TO 


MOTHERS 


IN EACH EMPLOYMENT CATEGORY 


(A COST 


WORK SHEET) 








Program options 






I 


II 


III 


IV 


V 








Added 




Home health 








hours 


Home health 


and added 


Mothers’ employment status 


On-going 


Added 


and 


and added 


hours and 


program 


hours 


weeks 


hours 


weeks 


Remain at home 


900 


750 


675 


675 


615 


Employed between 
9 A.M.-3 P.M. onb 

Employed within hours 


150 


75 


60 


105 


90 


8 A.M.-6 P.M. but not exclu- 
sively during 9 A.M.-3 P.M.. . 


360 


600 


690 


645 


720 


Employed hours other than 












8 A.M.-6 P.M 


90 


75 


75 


75 


75 


Total 


1,500 


1,500 


1,500 


i Finn 


i Finn 




1 >DUU 


1 jOUU 



Program V is a provision of school 
activities from 8 a.m.-6 p.m. and also 
home health services for 52 weeks. 

Effectiveness 

To compare these programs, we 
ask: How many mothers would have 
an improved choice of more satisfac- 
tory employment? Some mothers 
clearly have children below school age 
as well as children of school age and 
accordingly could not benefit from the 
program options as they are set forth. 
Some others are not able to work out- 
side the home for other reasons. We 
then ask: How many mothers from 
among those with an improved choice 
would actually choose to work outside 
the home? What would be added to 
family income? 

Information would have to be gath- 
ered in our hypothetical community 
to determine how many mothers 
would be affected and their potential 
earnings. Table 2 displays the infor- 
mation obtained on mothers’ employ- 
ment status under each program. 



A total of 1,000 mothers is assumed 
for the children five years of age and 
over that are supervised by the school. 
Under the on-going program 600 of 
the mothers are assumed to remain at 
home and 400 to work outside the 
home. For each of the program op- 
tions illustrative figures are displayed 
for the employment status of the 1,000 
mothers. (We assume for the immedi- 
ate period no net movement in or out 
of the community.) 

Table 3 presents illustrative figures 
on added earnings that would be 
made possible by adoption of one or 
the other of the several program op- 
tions. These illustrative figures re- 
flect the assumed increases owing to 
(a) more mothers working and (b) 
more mothers who are enabled to take 
on full-time or more responsible jobs. 
It also includes saving in child care 
expenses which are being incurred by 
those mothers who work under the 
on-going school program. (Data would 
have to be developed that permit an 
estimate of such expenses.) 
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Costs of Programs 

For each program option, cost fig- 
ures that include the full cost impli- 
cations of the additional child care 
services would be estimated. Various 
methods of cost calculation may be 
applied. For the hypothetical com- 
munity with its distribution of moth- 
ers and pupils we have developed cost 
estimates based on on-going school 
costs by pupil. For the program op- 
tions that include home health ser- 
vices as well as extended school super- 
visory activities cost calculations are 
based on total costs per home health 
service worker. 

For example, there are assumed to 
be 1,500 pupils who are potential 
users of each program. The use pupils 
make of a program is likely to be de- 
termined by the employment status 
of the mothers. 

Table 4 shows an illustrative dis- 
tribution of pupils by the employment 
status of their mothers. (It has been 
assumed that there are li/ 2 pupils for 
every mother.) 

How many of those potential users 
of the program offerings would use 



them? Table 5 shows the illustrative 
percentage of pupils in each category 
who are expected to make use of the 
program. (Data for this type of dis- 
play could be collected by means of 
a survey or estimated by reference to 
the experience of other communities 
with similar types of programs.) 

A “market-type” study on demand 
for home health service combined 
with school absenteeism data or health 
statistics would provide the informa- 
tion needed to make a calculation of 
requirements. For the table presented 
here we have filled in hypothetical 
numbers by assumption. For example, 
we have assumed that mothers work- 
ing full time would make most use of 
day care programs, mothers working 
part of the day would choose to make 
less use of the facilities, and mothers 
remaining at home during the time 
the facilities were offered would make 
some but not much use of them. 

In the illustrative figures an as- 
sumed peak load demand for home 
health services was taken into account 
because of the seasonal pattern of ill- 
ness. For example, it was assumed 



TABLE 5.— 


PERCENT 


OF 


PUPILS 


OPTING FOR EACH PROGRAM 












Program options 












I 


II 


III 




IV 






V 








Day 

care 

40 

weeks 


Day care 


Flnv Hnmp- 


Day care 


Homemaking 


Mothers’ employment 
status 


On- 
going 
program i 


40 

weeks 


12 

weeks 


care 

40 

weeks 


making 

40 

weeks 


40 

weeks 


12 

weeks 


40 

weeks 


12 

weeks 


Remain at home 

Employed between 9 A.M. 


109% 


10% 


10% 


20% 


10% 


1% 


10% 


20% 


1% 


1% 


and 3 P.M. only 

Employed within hours 
8 A.M.-6 P.M. but not 
exclusively during 


100 


20 


20 


60 


20 


r> 


20 


60 


5 


r> 


9 A.M.-3 P.M 

Employed hours other 


100 


60 


60 


70 


60 


5 


60 


70 


5 


5 


than 8 A.M.-6 P.M. , . , 


100 


10 


10 


20 


10 


1 


10 


20 


1 


1 
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TARIE 6 — ILLUSTRATIVE number of pupils opting for each program — 

A COST WORK SHEET 











Program options 












I 


II 


III 






IV 




V 










Added 


Added hours 
and weeks 


Home health 
and 

added hours 


Home health and added 

lv /Mivr nn/I Tirnnlr c 






hours 


Day- 

care 


Home 












On- 

going 

program 


day 

care 

40 

weeks 


Day care 


health 


Day care 


Home health 


Mothers’ employment 
status 


40 

weeks 


12 

weeks 


40 40 

weeks weeks 


40 12 

weeks weeks 


40 

weeks 


12 

weeks 


Remain at home 




75 


68 


136 


68 


7 


62 


124 


6 


6 


Employed between 9 A.M. 
and 3 P.M. only 




15 


12 


36 


21 


5 


18 


54 


5 


5 


Employed between 8 A.M. 
and 6 P.M. but not 
exclusively 9 A.M. 
and 3 P.M 




360 


414 


483 


487 


32 


432 


504 


36 


36 


Employed hours other 
than 8 A.M. and 6 P.M. 




8 


8 


16 


8 


I 


8 


16 


1 


1 


Total 




458 


502 


671 


484 


45 


520 


698 


48 


48 



that a maximum of 5 percent of 
pupils would stay in at home at any 
one time. After taking account of ill- 
ness of siblings, it further was as- 
sumed that perhaps something over 
3 percent of the homes would poten- 
tially need home health care at any 
one time. 

It also was assumed that almost all 
of the working mothers would make 
use of home health services when 
needed, and that 20 percent of non- 
working mothers would make use of 
these services. 

Based on the illustrative percentage 
of pupils in each category of “employ- 
ment status of mothers” the number 
of pupils who would use the alterna- 
tive program offering may be esti- 
mated. The hypothetical figures are 
shown in Table 6. 

After establishing the numbers of 
users of the program offering, the full 



added cost per user must be deter- 
mined. Facility costs that are “fixed” 
and that would not be affected by the 
additional services would not be 
counted — school facility costs, for ex- 
ample. New equipment needs, how- 
ever, would be included as would 
additional expenditures for staffing, 
fringe benefits, school meals, and so 
forth. 

In setting down the estimates of 
dollar cost for the program option in 
our hypothetical case in Table 7 it 
was assumed that the additional cost 
for each program was as follows: 

The 40-week program of 8-6 care was $700 
per pupil. 

The 12-week program of 8-6 supervision 
during the school recess period was $420 per 
pupil. 

The home health costs for 40 weeks was 
$2,800 per worker. 

The home health costs for 12 weeks was 
$800 per worker. 
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It was assumed that one home 
health worker would be required per 
household needing home health care. 

An analysis would require an as- 
sessment of the availability of person- 
nel and other resources to carry out 
each of the program options as well 
as of program benefits. 

Summary of Effectiveness 
and Costs 

We have defined several measures 
on the basis of which to judge the 
progress made by alternative pro- 
grams toward satisfying the two de- 
fined objectives, namely: 

1. To give mothers more choice in 
satisfactory employment 

2. To enhance family income. 

Two of the measures are dollar 

criteria and three are psycho-social 
criteria. The progress that woidd be 
made under each option is shown in 
Tables 2 and 3 and is displayed along 
with estimated costs in Table 8. 

Clearly the types of data shown in 
Table 8 are more informative than 
the usual staff materials developed for 



policy officials. Moreover, qualifica- 
tions on such informational materials 
help to put in perspective the analysis 
carried out. 

In limiting the analysis, for exam- 
ple, to the employment of mothers as 
the defined objective, we have left out 
a host of important benefits that re- 
late to other purposes. Included 
among those omitted are: 

1. Benefits in learning, by children 

2. Benefits to family and commun- 
ity from extra supervision of children 

3. Reduction in community costs 
attributable to saving in public wel- 
fare, in juvenile delinquency, and un- 
treated illness 

4. Advances from poverty 

5. Secondary expansion brought to 
the communities’ economy through 
the mothers’ expenditure of their ex- 
tra income. 

We also limited the analysis to a 
range of five options. On completion 
of the initial analysis, a decision may 
be reached as follows. The cut-off 
point of alternatives was too soon. 
Additional options need to be consid- 



TABLE 7.— ILLUSTRATIVE ESTIMATES OF PROGRAM COSTS 



Program options 





I 


ii 


III 


IV 


V 


Additional costs 


On-going 

program 


Added 

hours 


Added 

hours 

and 

weeks 


Home health 
and added 
hours 


Home health 
and added 
hours and 
weeks 


DAY CARE 

Additional costs per pupil 

$700 for 40 weeks 

$420 for 12 weeks 




$321,000 


S35 1,000 
282,000 


$339,000 


S364.000 

293,000 


HOME HEALTH 

(Costs per home health worker) 

$2,800 for 40 weeks 

S800 for 12 weeks 








120,000 


128,000 

38,000 


Total 




S32 1,000 


S633.000 


S459.000 


$823,000 
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tarlp ft. — ILLUSTRATIVE SUMMARY INFORMATION PROVIDED TO 

DECISION MAKER 



Program options 







i 


II 


III 


IV 


V 










Added 




Home health 










hours 


Home health 


and added 






On-going 


Added 


and 


and added 


hours and 




Evaluation criteria 


program 


hours 


weeks 


hours 


weeks 






Effectiveness to satisfy employment and income objectives * 


1 . 


Mothers with improved 
choice ' 


0 


800 


800 


800 


800 


2. 


Mothers previously working 












3. 


obtaining more satisfactory 

employment 

Mothers newly employed 


0 

0 


80 

100 


100 

150 


100 

150 


120 

190 


4. 


Added family income and 
saving in child care costs 


0 


$300,000 


$615,000 


$530,000 


$901,000 










Added costs 




5. 


Dollars of additional total 
costs in year a 


0 


$321 


$633 


$459 


$823 



Other purposes served 



Improved learning of children . . 
Reduced social problems result- 
ing from child neglect 

Reduced public dependency and 

poverty 

Higher community income 
(both present and in future). 

* From Tables 2, 3, and 7. 



ered, for example, reduced age at 
which child care is provided, or home 
health services without other day-care 
facilities. 

Ideally, analyses or programs of this 
nature should not be carried out ex- 
clusively for different levels of effort 
in only one year, but on a multi-year 
basis. Costs of the program may be 
highest in the first year while the pro- 
gram is being set up, whereas benefits 
may be lower in the first year. For 
example, not all mothers who seek em- 



ployment may find it immediately. 
Furthermore, as the mothers work a 
longer period and gain experience, 
their average income may rise. 

The purpose of this sample cost / 
benefit case has been to display some 
of the informational materials that 
would result from analysis. A full- 
scale analysis would involve many 
extra steps, but the same type of 
analytical approach would be applied 
and similar categories of critical in- 
formation would be used. 
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